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Abstract 

In this article, the logic rule ensembles approach to supervised learning 
is applied to the unsupervised or semi-supervised clustering. Logic rules 
which were obtained by combining simple conjunctive rules are used to 
partition the input space and an ensemble of these rules is used to define 
a similarity matrix. Similarity partitioning is used to partition the data 
in an hierarchical manner. We have used internal and external measures 
of cluster validity to evaluate the quality of clusterings or to identify the 
number of clusters. 

Keywords & Phrases: ensemble learning, clustering, biological annotation, 
logic rule, random projection 

1 Introduction 

Cluster analysis is used to identify groupings of individuals in populations based 
on data. The identified groups partition the data space and are called clusters. 
The lack of a universal definition of a cluster, and its task or data dependent na- 
ture has resulted in publication of a very large number of clustering algorithms. 

The approach taken in this paper for clustering is closely related to the 
decision tree, rule and logic rule ensemble approaches to the supervised learning 
problem. The logic rule ensembles combine the propositions about individual 
input variables with with logic operators "and" , "or" and "not" and use them as 
input variables (pQ) to estimate the target variable. Each rule takes the values in 
{0, 1} and partitions the sample space into two parts. We use the data as input 
variables and random projections of this data ([IT], [2]) as target variables and 
combine the rules learned from these supervised problems into one clustering 
using the cluster based similarity partitioning framework of Ghosh et. al. ([21]). 

In the following section, we review how the importance sampling learning 
ensemble (ISLE) algorithm ([H]) can be used to generate an ensemble of con- 
junctive rules and the more familiar and concise logic rules for the supervised 
learning problem. This approach is then adopted to the clustering problem in 
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Section 3. In Section 4, we illustrate the ensemble clustering algorithm and 
compare it to several popular clustering algorithms. 

2 Supervised Learning with (Logic) Rules 

Given a learning task and a relevant data set, we can generate a set of models 
from a predetermined model family. Bagging bootstraps the training data set 
[3] and produces a model for each bootstrap sample. Random forest ([TJJ [5]) 
creates a diverse set of models by randomly selecting a few aspects of the data set 
while generating each model. AdaBoost [10] and ARCing |4j iteratively build 
models by varying case weights (up-weighting cases with large current errors 
and down- weighting those accurately estimated) and employs the weighted sum 
of the estimates of the sequence of models. There have been few attempts to 
unify these ensemble learning methods. One such framework is the ISLE due 
to Popescu & Friedman [TT] . 

We are to produce a regression model to predict the continuous outcome 
variable y from p vector of input variables x. We will generate models from a 
given model family & = {f(x, 9) : 9 e 0} indexed by the parameter 9. The 
final ensemble models considered by the ISLE framework have an additive form: 

M 

F{x)^w Q + Y / w J f(x 1 9 J ) (1) 

where {f(x, 0j)}jLi are Dase learners selected from ISLE uses a two-step 
approach to produce F(x). The first step involves sampling the space of possible 
models to obtain {9j}jL 1 . The second step proceeds with combining the base 
learners by choosing weights {wj}jL in ((T|). 

The pseudo code to produce M models {f(x, 6j)}jL 1 under ISLE framework 
is given below: 

Algorithm 2.1: ISLE(M, v, rj) 

F (x) = 0. 
for j=l to M 

{(Cj,8j) = argmin]T\ s , } Lfa, -Fj-i(xi) + cf{ Xi ,9)) 
(c,e) 
T j (x) = f(x,9 j ) 
F j (x)=F j ^ 1 (x) + uc j T j (x) 
return ({T 3 {x)}f =1 and F M {x).) 

Here L(., .) is a loss function, Sj(rf) is a subset of the indices {1,2, ... ,n} 
chosen by a sampling scheme 77, < v < 1 is a memory parameter. 

The classic ensemble methods of Bagging, Random Forest, AdaBoost, and 
Gradient Boosting are special cases of ISLE ensemble model generation proce- 
dure [20] . In Bagging and Random Forests the weights in [1] are set to predeter- 
mined values, i.e. wq — and Wj — -A> for j = 1,2, ... , M. Boosting calculates 
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these weights in a sequential fashion at each step by having positive memory 
estimating Cj and takes Fm{x) as the final prediction model. 

Friedman & Popescu recommend learning the weights {wj}jL Q using 

lasso [12]. Let T = (T J -(x i ))" =1 ^ =1 be the n x M matrix of predictions for the n 
observations by the M models in an ensemble. The weights (wq,w — {w m }^- ) 
are obtained from 

M 

w = argmin(y - w l n - Tw)'(y - w l n - Tw) + A ^ |io m |. (2) 

w . 

m — 1 

A > is the shrinkage operator, larger values of A decreases the number of 
models included in the final prediction model. The final ensemble model is 
given by 

M 

F{x)=w +J2w m T m (x). (3) 

m— 1 

Given a set of decision trees, rules can be extracted from each of these trees 
to produce a collection of rules. Let R = (rk{xi)) i=lk=1 be the n x K matrix 
of rules for the n observations by the K rules in the ensemble. The rulefit 
algorithm of Friedman & Popescu [H] uses the weights {wq,w = {wk}^ =0 ) that 
are estimated from 

K 

w = argmin(y - u; l„ - Rw)'(y - w a l n ~ Rw) + A^ \w k \ (4) 

fe=i 

in the final prediction model 

K 

F(x) = w +^2w k r k (x). (5) 

fe=l 

A conjunctive rule r(x) — nf=i I( x i ^ s i) can a ^ so be expressed as a logic 
rule (also called Boolean expressions and logic statement) involving only the 
A ("and") operator. In general, a logic statement is constructed using the 
operators A ("and"), V ("or") and c ("not") and brackets. An example simple 
logic rule is 

l(x) = [I(xi e si) V I c {x 2 e s 2 )] A I(x 3 G s 3 ). 

It should be noted that the representation of a logic rule in general is not 
unique. However, it can be shown that all logic rules can be expressed in dis- 
junctive normal form where we only use V combinations of A terms. Coupling 
this with the De Morgan's laws we see that all logic rules can actually be writ- 
ten as conjunctive rules. However, the additional "or" operator in logic rules 
in disjunctive normal form provide consolidation of interchangeable rules and 
therefore provide more precise and interpretable results. 

Given the logic rules {h(x), {h(x), . . . , {l L (x)}, let S = (lk(xi))™ =lk=1 be 
the nx K matrix of logic rules for the n observations by the L logic rules in the 
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ensemble. We propose using the weights (wq,w — {wk} k=0 ) that are estimated 
from 



L 



w 



= argmin(y - w l n - Sw)'(y - w l n - Sw) + X^\w k \ (6) 

fc=i 

in the final prediction model 

L 

F(x) =w + J2 w kh(x). (7) 

k=l 

We call this the logic rule ensemble. 

The absolute values of the standardized coefficients 

{Impk = \w k \y/v k (l - v k )}, k = 1,2, ...,L 

can be used to evaluate the importance of a rule ([I2])- Here v k is the support 
of the rule k and defined as v k — X)"=i s k{ x i)/ n - A measure of importance for 
each variable can be obtained as the sum the importances of rules that involve 
that variable. 



3 (Logic) Rule Ensemble Clustering 

The main difference between the supervised learning and unsupervised learn- 
ing is the existence of a target variable in the former. The sample partitioning 
approaches discussed in the previous section partition the sample space into 
clusters. Each rule defines a clustering of the sample space into two compo- 
nents which give a good segregation of the target variable. When we are not 
provided with a target variable, we construct our target variables by mapping 
the input variables. Each of these target variables can be used to extract several 
interesting rules and overall cluster rules are obtained from combining the rules 
for many target variables into an ensemble distance matrix. The details of the 
procedure are described below. 

Algorithm 3.1: SS-ISCA(X, Y, M, m, v) 

Ri : A random projection of Y 
for j = 1 to M 

'Generate m logic rules {h(x)}gLi to estimate Rj from X: 
Sj(x) <= {k(x)}f =1 
do I T(X) <= {SjixMULx _ 

Rj+i : A random projection of Y 
^Rj+i <={I- vP T ( X ))Rj + x 
return (T(X)) 

Here Pt(x) is the projection matrix on to the space spanned by the columns 
of T(X) calculated as T{X)[T(X)'T(X)]-T(X). The time it takes to calculate 
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this matrix grows quickly as the algorithm advances in its stages. For high 
dimensional datasets, where M or m should be selected large, this makes the 
ISCA algorithm to be slow. One simple strategy is to apply lasso regression to 
select rules at each stage j and therefore reducing m. Another strategy is to 
sample the columns of T(X) to calculate Ptix) a t each stage, and in this case 
v is used as the sampling proportion. We have used both of these techniques 
successfully for high dimensional clustering problems. Of course, if the memory 
parameter v is set to zero then we do not have to worry about calculating Pt(x) ■ 
In this case the ISCA algorithm is parallelizable. 

From T(X) nxr , a similarity matrix can be obtained as S(X) — T(X)T(X)' jr. 
Note that T(X) is a sparse matrix. The zjth element of S(X) is the percentage 
of times the ith and the jth observations fire the same rules. From the similarity 
matrix S(X), we calculate a distance matrix D(X). The clustering of the ob- 
servations is accomplished by applying a distance based hierarchical clustering 
algorithm (|TB]) to the rule based distance matrix D(X). The method we have 
used to combine the clusters from many rules is referred to as the cluster based 
similarity partitioning in Ghosh et. al. ([21 ). 

In distance based hierarchical clustering, first, each object is assigned to its 
own cluster. At each consecutive stage the two most similar clusters are joined 
until there is a single cluster. The distances between clusters are recalculated 
at each stage by a linkage criterion such as single-linkage, complete linkage or 
average linkage. In our illustrations in the following section, we have uniformly 
used the average linkage criterion for combining clusters. 

For generation of rules useful for ensemble clustering when there are no 
target variables, we modify the semi-supervised rule generation algorithm in 13.11 
as follows: 

Algorithm 3.2: ISCA(A, M, m, v) 

Ri : A random projection of X 
for j = 1 to M 

'Generate m logic rules to estimate Rj from X: 

Sjix) ^ {l e (x)}f =1 
do I T{X) <= {Sjixi)}?^ 

Rj+i : A random projection of X 

<= {I — vPt(x))Rj+i 
return (T(Xj) 

Hierarchical clustering produces nested clusterings of the data set. In order 
to determine the final clustering, the number of clusters has to be determined. 
When the clustering problem is accompanied by external class labels or exter- 
nal benchmarks and the number of clusters is known, the quality of a cluster 
can be measured by Rand measure (R) ([IS]), Jaccard index (J) ([7]), Fowlkes 
Mallows index (FM) ([5]), Wallace indices (W01, W10), etc... Otherwise inter- 
nal clustering quality measures like Silhouette (silhouette) ([12])) Dunn Index 
(dunn) ( 8 ), Connectivity (connect) ([6]) can be used to identify the number 
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of clusters. These measures can also be used to compare the quality of several 
clusterings. The article by Handl et al. ([13]) provides an excellent overview of 
cluster validation measures. 

4 Illustrations 

We illustrate the clustering approaches in four real data sets. In addition to the 
SS-ISCA and ISCA approaches, we employ existing methods like model based 
clustering (Mclust), partitioning around medoids (PAM), divisive analysis clus- 
tering (DIANA) and random forest clustering (RF). The different clusterings are 
compared using the internal measures (Silhouette, Dunn index, Connectivity) 
or using the external measures (Rand measure, Fowlkes Mallows index, Wallace 
indices, Jaccard index). In the case where a supervisory target variable was 
available and there were only two clusters, we also provided the p values from 
comparing the means for the target variable in these clusters. 

Example 4.1. (Fisher's Iris Data Set) The results of clustering the Fisher's 
Iris data set are displayed in Table [7J The existing data labels are used to cal- 
culate the internal validity measures. We have also provided internal measures 
of cluster quality. ISCA algorithm is uniformly the best according to the inter- 
nal measures. With respect to the external validation measures ISCA algorithm 
ranks second after the model based clustering. 



Table 1: (Fisher's Iris Data Set, Unsupervised Clustering) Internal and external 
measures of cluster validity. ISCA algorithm is uniformly the best or second best 
according to these measures. ** and * are used to mark the best and the second 
best clusterings correspondingly. 





Internal 




External 










silhouette 


dunn 


connect 


R 


FM 


W01 


W10 


J 


ISCA 


0.55** 


0.12** 


7.40** 


0.89* 


0.83* 


0.86* 


0.81* 


0.71* 


RF 


0.48 


0.03 


23.26 


0.71 


0.66 


0.79 


0.54 


0.47 


PAM 


0.55** 


0.10 


10.09* 


0.88 


0.82 


0.84 


0.81 


0.70 


DIANA 


0.54 


0.11* 


12.43 


0.86 


0.80 


0.81 


0.78 


0.66 


Mclust 


0.50 


0.07 


14.18 


0.96** 


0.94** 


0.94** 


0.93** 


0.88** 



Example 4.2. (FHB Data Set, Semi- Supervised Clustering) FHB is a plant 
disease caused by the fungus Fusarium Graminearum and results in tremen- 
dous losses by reducing grain yield and quality. In addition to the decrease in 
grain yield and quality, another damage due to FHB is the contamination of the 
crop with mycotoxins. Therefore, breeding for improved FHB resistance is an 
important breeding goal. The Fusarium Head Blight (FHB) data set contains 
information on 2251 markers, along with the FHB and DON levels for 622 elite 
barley lines. The data is available from the author upon request. A very detailed 
explanation of this data set is given in \ 15}/ . We would like to segregate the 
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622 barley lines into two groups, low resistance and high resistance lines. The 
results from clustering this data using different approaches are summarized by 
the box plots in Figured SS-ISCA clearly gives the best segregation of the FHB 
variable among other clusterings which we measure by the p value corresponding 
to the two sample t test for comparing group means. Some internal measures 
for cluster quality for clusterings by different clustering approaches are provided 
in Tabled 



ISCA 



SS-ISCA 



Mclust 



Figure 1: (FHB Data Set, Semi-Supervised Clustering) The FHB data set con- 
tains information on 2251 markers, along with the FHB and DON levels for 622 
elite barley lines, p values from the t tests corresponding to different cluster- 
ing approaches indicate that the SS-ISCA and ISCA produce groups that are 
different from each other in terms of the mean FHB. 



Example 4.3. (Stem Rust Data Set) The stem rusts is a disease affecting cereal 
crops. Crop species which are affected by the disease include wheat, barley and 
triticale. We had estimated breeding values of stem rust resistance for 374 lines 
of wheat. In addition 1624 markers were available for these lines. The box 
plots in Figure [H compare the stem rust resistance for the groups from several 
clustering algorithms. The p-values from the two sample t test are also provided. 
The best segregation is obtained again by the ISCA approach. 
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Table 2: (FHB Data Set, Semi-Supervised Clustering) SS-ISCA and ISCA clus- 
terings outperform other clusterings. 





silhouette 


dunn 


connect 


SS-ISCA 


0.108 


0.419** 


44.450** 


ISCA 


0.114* 


0.344* 


67.763* 


RF 


0.092 


0.344* 


111.410 


PAM 


0.126** 


0.160 


206.743 


Mclust 


0.114* 


0.317 


126.918 



ISCA 



~i 1 

1 2 



RF 



PAM 



DIANA 



n r~ 

1 2 



n !~ 

1 2 



"i r 

1 2 



Mclust 



~l !~ 

1 2 



p-val=0 



p-val=0.004 



p-val=0.335 



p-val=0.165 



p-val-0.003 



Figure 2: (Stem Rust Data Set, Semi-Supervised Clustering) The best segrega- 
tion is obtained by ISCA approach. 
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Example 4.4. (Mouse Data Set, Learning the Number of Clusters) We use 
data from an Afyymetrix microarray experiment comparing gene expression of 
mesenchymal cells from two distinct families, neural crest and mesoderm de- 
rived. The dataset consists of 147 genes and expressed sequence tags (EST) 
which were determined to be significantly differentially expressed between the 
two cell groups. For further description of the dataset and the experiments the 
reader is referred to Bhattacherjee et al. (2007). The internal measures of clus- 
ter quality is displayed for 2 to 30 groups clustering by ISC A in Tabled The 
optimal number of clusters is determined to be two using the connectivity and 
silhouette width. Although the Dunn Index increases almost uniformly after 3 
clusters and attains very high levels after 10 or more clusters, there is indication 
that 2 groups provides a reasonable clustering. 

5 Conclusion 

We have discussed how we can use an ensemble of logic rules for unsupervised 
and semi-supervised cluster learning. Our examples show that the approaches 
introduced herein are promising, they produce high quality clusters as indicated 
by many internal and external measures of cluster quality. 
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Figure 3: (Mouse Data Set, Learning the Number of Clusters) The internal 
measures of cluster quality is displayed for 2 to 30 groups clustering by ISCA. 
Hierarchical clustering with two clusters performs the best in general. 
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